Linux Programming - Dive Into the Select Model

Overview

This last article briefly described the select model and the system call usage. In this article let’s dive deeper into it.

The key of the select model: fd_set

The key to understanding the select model is to understand that fd_set. fd_set type is actually an array of long type. For convenience, suppose fd_set is 1 byte in length. Each bit in fd_set can correspond to a file descriptor (fd), then a 1-byte long fd_set can correspond to a maximum of 8 fds.

1. Execute fd_set set; FD_ZERO(&set); The set is expressed in bits as 0000,0000.

2. If fd = 5, execute FD_SET(fd, &set); then set becomes 0001,0000 (the fifth position is 1)

3. If fd = 2 and fd = 1 are added, then set becomes 0001,0011

4. Execute select(6, &set, 0, 0, 0) to block wait

5. If a readable event occurs on both fd = 1 and fd = 2, then select stops blocking wait, and set becomes 0000,0011. Note: fd = 5 is cleared as no event occurs on that bit.

Based on the above discussion, the characteristics of the select model can be easily derived as:

1. The number of file descriptors that can be monitored depends on the value of sizeof(fd_set). On my computer, sizeof(fd_set) = 512. Each bit represents a file descriptor, then the largest file descriptor supported on my computer is 512 * 8 = 4096. The upper limit of the value depends on the value of FD_SETSIZE. This value is fixed after compiling the Linux kernel.

2. When a fd is added to the select, an array fd_set is used to store the fd.

3. After select returns (stops blocking the I/O), by calling FD_ISSET, we can check if the bit that represents the file descriptor (fd) returned by the I/O operation is set, then we can know if the I/O event occured.

4. Before calling select, we need to empty the fd_set by FD_ZERO first, then use FD_SET to set the fd in a bit of the fd_set. Note after select returns, all the fds that have no associated events occur will be cleared. So we need to use FD_SET to set the fd in the fd_set before that each select call. Here is an example:
   

1
2
3
4
5
6
7
8
fd_set rd;
int fd;
FD_ZERO( & rd);
while (1) {
  FD_SET(fd, & rd);
  ret = select(1, & rd, NULL, NULL, NULL);
  ...
}

You see in the while loop, FD_SET is called each time before calling select.

Let’s look at some more complicated examples of select, which are commonly seen in production.

Example 1: Handling out-of-band data with select

In network programs, select can handle only one type of abnormal situation: out-of-band data is received on the socket.

What is out-of-band data? Out-of-band data, sometimes also called expedited data, means one of the two parties in a connect has something important and wants to notify the other party quickly.

Usual data are put in a transmission queue. These data are referred to as “in-band” data. For “out-of-band” data, they need to be sent before any “in-band” data. Then we can know:

1. Out-of-band data is designed to have higher priority than normal data.

2. Out-of-band data is mapped into an existing connection, rather than using another connection between the client and server.

Usually, select is used to receive “in-band” data. However, our server needs to receive both “in-band” data and “out-of-band” data. Here is an example to use select to handle both “in-band” data and “out-of-band” data:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
#include <stdio.h>

#include <stdlib.h>

#include <string.h>

#include <unistd.h>

#include <sys/types.h>

#include <sys/socket.h>

#include <sys/time.h>

#include <netinet/in.h>

#include <arpa/inet.h>

#include <fcntl.h>

int main(int argc, char * argv[]) {
  if (argc <= 2) {
    printf("usage: ip address + port numbers\n");
    return -1;
  }

  const char * ip = argv[1];
  int port = atoi(argv[2]);

  printf("ip: %s\n", ip);
  printf("port: %d\n", port);

  int ret = 0;
  struct sockaddr_in address;
  bzero( & address, sizeof(address));
  address.sin_family = AF_INET;
  inet_pton(AF_INET, ip, & address.sin_addr);
  address.sin_port = htons(port);

  int listenfd = socket(PF_INET, SOCK_STREAM, 0);
  if (listenfd < 0) {
    printf("Fail to create a listen socket!\n");
    return -1;
  }

  ret = bind(listenfd, (struct sockaddr * ) & address, sizeof(address));
  if (ret == -1) {
    printf("Fail to bind socket!\n");
    return -1;
  }

  // Set the maximum number of listening fds to 5
  ret = listen(listenfd, 5);
  if (ret == -1) {
    printf("Fail to listen socket!\n");
    return -1;
  }

  struct sockaddr_in client_address; // The IP address of the client
  socklen_t client_addr_length = sizeof(client_address);
  int connfd = accept(listenfd, (struct sockaddr * ) & client_address, & client_addr_length);
  if (connfd < 0) {
    printf("Fail to accept!\n");
    close(listenfd);
  }

  char buff[1024]; // Data receive buffer
  fd_set read_fds; // Read file descriptor for in-band data
  fd_set exception_fds; // Exception file descriptor for out-of-band data

  // Empty these file descriptors
  FD_ZERO( & read_fds);
  FD_ZERO( & exception_fds);

  while (1) {
    memset(buff, 0, sizeof(buff));
    // Set the fd_set bit before each select call
    FD_SET(connfd, & read_fds);
    FD_SET(connfd, & exception_fds);

    ret = select(connfd + 1, & read_fds, NULL, & exception_fds, NULL);
    if (ret < 0) {
      printf("Fail to select!\n");
      return -1;
    }

    // Check if we received any data (read event)
    if (FD_ISSET(connfd, & read_fds)) {
      // If so, read the data.
      ret = recv(connfd, buff, sizeof(buff) - 1, 0);
      if (ret <= 0) {
        break;
      }

      printf("Got %d bytes of normal data (in-bound): %s \n", ret, buff);

    } else if (FD_ISSET(connfd, & exception_fds)) // Exception event
    {
      // Receive the data as out-of-band (MSG_OOB) 
      ret = recv(connfd, buff, sizeof(buff) - 1, MSG_OOB);
      if (ret <= 0) {
        break;
      }

      printf("Got %d bytes of exception data (out-of-band): %s \n", ret, buff);
    }

  }

  close(connfd);
  close(listenfd);

  return 0;
}

From the example above, you can see we distinguish “in-band” data and “out-of-band” data into two different file descriptors (fd), and use FD_ISSET to check which event occurs. When read “out-of-band” data, we need to pass MSG_OOB into recv system call. Then the system is able to receive “out-of-band” data before “in-band” data.
Example 2: Handling multiple clients in the socket programming
One of the biggest advantages of the select model is that we can handle multiple socket I/O requests at the same time in one thread.

In network programming, when it comes to multiple clients accessing the server, the first way we think of is to fork multiple processes to handle each client connection separately. But, this way is very resource consuming. With select, we are able to handle multiple clients without fork.

Let’s see a concrete example.

Server side code

The code of the server side is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
#include <stdio.h>

#include <stdlib.h>

#include <unistd.h>

#include <sys/types.h>

#include <sys/socket.h>

#include <sys/time.h>

#include <sys/ioctl.h>

#include <netinet/in.h>

int main() {
  int server_sockfd, client_sockfd;
  int server_len, client_len;
  struct sockaddr_in server_address;
  struct sockaddr_in client_address;
  int result;
  fd_set readfds, testfds;
  // Create the server side socket
  server_sockfd = socket(AF_INET, SOCK_STREAM, 0);
  server_address.sin_family = AF_INET;
  server_address.sin_addr.s_addr = htonl(INADDR_ANY);
  server_address.sin_port = htons(8888);
  server_len = sizeof(server_address);
  bind(server_sockfd, (struct sockaddr * ) & server_address, server_len);
  // Set the maximum number of listening fds to 5
  listen(server_sockfd, 5);
  FD_ZERO( & readfds);
  // Put the fd of the socket to the fd_set
  FD_SET(server_sockfd, & readfds);
  while (1) {
    char ch;
    int fd;
    int nread;

    // As select will modify the fd_set readfds
    // So we need to copy it to another fd_set testfds
    testfds = readfds;
    printf("Server is waiting\n");

    // Block indefinitely and test file descriptor changes
    // FD_SETSIZE：the system's default number of maximum file descriptors
    result = select(FD_SETSIZE, & testfds, (fd_set * ) 0, (fd_set * ) 0, (struct timeval * ) 0);
    if (result < 1) {
      perror("Failed to select!\n");
      exit(1);
    }

    // Loop all the file descriptors
    for (fd = 0; fd < FD_SETSIZE; fd++) {
      // Find the fd that associated event occurs
      if (FD_ISSET(fd, & testfds)) {
        // Determine if it is a server socket
        // if yes, it indicates that the client requests a connection
        if (fd == server_sockfd) {
          client_len = sizeof(client_address);
          client_sockfd = accept(server_sockfd,
            (struct sockaddr * ) & client_address, & client_len);
          // Add the client socket to the collection
          FD_SET(client_sockfd, & readfds);
          printf("Adding the client socket to fd %d\n", client_sockfd);
        }
        // If not, it means there is data request from the client socket
        else {
          // Get the amount of data to nread
          ioctl(fd, FIONREAD, & nread);

          // After the client data request is completed
          // The socket is closed and the corresponding fd is cleared
          if (nread == 0) {
            close(fd);
            // Remove closed fd
            // (from the unmodified fd_set readfds)
            FD_CLR(fd, & readfds);
            printf("Removing client on fd %d\n", fd);
          }
          // Processing the client data requests
          else {
            read(fd, & ch, 1);
            sleep(5);
            printf("Serving client on fd %d\n", fd);
            ch++;
            write(fd, & ch, 1);
          }
        }
      }
    }
  }

  return 0;
}

Client side code

The code of the client side is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
#include <stdio.h>

#include <stdlib.h>

#include <unistd.h>

#include <sys/types.h>

#include <sys/socket.h>

#include <sys/time.h>

#include <netinet/in.h>

#include <arpa/inet.h>

int main() {
  int client_sockfd;
  int len;
  struct sockaddr_in address; // Server address structure family/ip/port
  int result;
  char ch = 'A';
  // Create the client socket
  client_sockfd = socket(AF_INET, SOCK_STREAM, 0);
  address.sin_family = AF_INET;
  address.sin_addr.s_addr = inet_addr("127.0.0.1");
  address.sin_port = htons(8888);
  len = sizeof(address);
  result = connect(client_sockfd, (struct sockaddr * ) & address, len);
  if (result == -1) {
    perror("Failed to connect to the server");
    exit(1);
  }
  // The first read & write
  write(client_sockfd, & ch, 1);
  read(client_sockfd, & ch, 1);
  printf("The first time: char from server = %c\n", ch);
  sleep(5);

  // The second read & write
  write(client_sockfd, & ch, 1);
  read(client_sockfd, & ch, 1);
  printf("The second time: char from server = %c\n", ch);

  close(client_sockfd);

  return 0;
}

Here are the steps to test

• Run a single server

• Run multiple clients (like 2 clients)

• Here is the result:

From the screenshots above, the server listens two clients in one thread, and both the clients received data from the server correctly.

Conclusion

The select model supports I/O multiplexing, so with the select model, we are able to handle multiple socket connections in a single thread. It is much better than the multi-threading solution.

However, it also has significant disadvantages.

1. The number of fds that can be monitored by a single process is limited, that is, the size of the listening port is limited. The limitation is related to the size of system memory. The specific number can be viewed by cat /proc/sys/fs/file-max. The default number for a 32-bit machine is 1024 for a 64-bit machine is 2048.

2. The socket is scanned linearly, that is, the polling method is used, and the efficiency is low. When there are more sockets, each select() must complete the scheduling by traversing all FD_SETSIZE sockets, no matter which socket is active. This will waste a lot of CPU time. If you can register a callback function for the socket and automatically complete related operations when they are active, then polling is avoided, which is exactly what epoll and kqueue does.

3. It is required to maintain a data structure used to store a large number of fds, which will cause the user space and kernel space to copy the structure with a large overhead.